US-Spray3D-MC3
Universität Stuttgart

VAST 2009 Challenge
Challenge 3 - Video Analysis

Authors and Affiliations:

Benjamin Höferlin, Universität Stuttgart, Benjamin.Hoeferlin@vis.uni-stuttgart.de [PRIMARY contact]

Markus Höferlin, Universität Stuttgart, Markus.Hoeferlin@vis.uni-stuttgart.de

Tool(s):           

The preprocessing of the video was done by a tool developed for this challenge in Matlab.

It analysis the moving objects of the video using a combined approach of optical flow computation and background subtraction.

Object tracking is done by a Kalman filter in screen space. Finally object properties are calculated based on manual camera calibration.

 

The video visualization and user interaction tool developed for this challenge is written in C++ and based on OpenGL.

The visualization applies the VideoPerpetuoGram methodology to get a summarization of the actions in the video. We combine keyframes at a

sparse interval of frames with the trajectories of moving objects.

 

For the visual analytic issue we implemented a real-time filter framework. Here we can apply different filters to the preprocessed object trajectories to omit the uninteresting ones.

 

The visual analytic process as the combination of the three parts: Computer Vision, Visualization and Interaction exploits the strengths of automatic video analysis and human cognitive abilities.

 

Video:

 

Vast_MC3.mov 

 

 

ANSWERS:

MC3.1: Provide a tab-delimitated table containing the location, start time and duration of the events identified above.

Video.txt

MC3.2:  Identify any events of potential counterintelligence/espionage interest in the video.  Provide a Detailed Answer, including a description of any activities, and why the event is of interest. 

Event Descriptions

High important meeting events

 

Location: 4                   Start Time: 7:01:13                   Duration: 3:59

A dark dressed man and a woman with a bright coat, bag and hat are talking to each other.

The man and the woman are also meeting in the video sequence at 7:31:05. This is a suspicious event,

since the woman is also involved in the briefcase exchange in 8:23:57.

 

Location: 4                   Start Time: 7:31:05                   Duration: 1:29

The already mentioned persons (woman & man from meeting at 7:01:13) are talking.

Maybe this is another meeting to negotiate time and location of the subsequent briefcase transfer.

 

Location: 2                   Start Time: 8:23:57                   Duration: 5:00

A black dressed man with a white briefcase meets the before mentioned white dressed lady. She carries a black briefcase.

After talking a few minutes they exchange their briefcases and leave the scenery. It looks like both people walked to their meeting place,

since there is no relevant change of the cars parked at the road. The woman leaves to the same direction as she came from.

 

Medium important meeting events

 

Location: 1                   Start Time: 0:45:20                   Duration: 0:15

A person fetches an object out of the red van and gives it to the other person.

The red van stood in this parking area since the first frame of the video capture.

After the transfer the red van is driven away. This event is interesting because of the object transfer.

 

Location: 4                   Start Time: 5:50:47                   Duration: 3:46

A man with white clothes and a bike waits for someone. Another person dressed in blue, arrives and they're talking to each other for a while.

This meeting is interesting because of its length and the fact that one of the men goes by bicycle – recall that the embassy is in walking range.

 

Low important meeting events

 

Location: 1                   Start Time: 9:13:49                   Duration: 17:43

A person is getting seated at 9:10:15 and another person joins at 9:13:19.

They're talking to each other for about 17 minutes. After that they're leaving and say goodbye.

 

Location: 2                   Start Time: 7:13:00                   Duration: 6:30

Two people talk to each other, one is clothed in red. After the meeting the red person is walking alone at 7:21:46.

 

 

Location: 2                   Start Time: 2:52:19                   Duration: 0:15

Two persons are talking to each other and are leaving in different directions after saying goodbye.

This indicates the end of a meeting, but especially there is nothing more suspicious.

 

Other events of interest

 

Location: 1                   Start Time: 0:08:03                   Duration: 2:41

A person leaves a house and carries a briefcase to a car (captured by all camera positions). Then he returns to the house.

This event gets interesting in the context of the briefcase exchange at Saturday.

But this man doesn’t seem to be the man directly involved in the exchange – he carries the briefcase with the left hand (in contrast to the man at 8:23:57).

 

Visual Analytic Process

The basic idea of our approach is to identify the encounter of people by their movement trajectories. The characteristics of these trajectories should help us to localize relevant parts of the sequence and though yield to a scalable method.

 

First, we separated the video sequence by an automated preprocessing step into the four locations, the camera captured. This preprocessing step also extracts the trajectories and calculates several additional properties.

The information include camera location, scene number, the temporal start and end positions, spatial positions of the tracks in pixel coordinates as well as spatial positions perspective projected to the ground, the mean speed and average direction.

 

For fast video exploration and filter interaction an easily operable visualization tool is necessary. Fig. 1 shows this tool and its visualization of the different camera positions as a 3D volume with time as third axis.

We can see a few keyframes and trajectories of detected objects. The blue bar indicates the time passed between two visualized scenes.

Between two scenes there is always a gap for at least 58 – 61 sec which originates from the camera movement. If we apply a filter this time may increase, as we see in the left column of Fig.1. For accelerated exploration, all scenes without any objects of interest are hidden.

 

1.png

Figure 1

 

Starting with the exploration of the trajectories it turns out, that many of them belong to cars driving on the street. Since these objects are not of interest to us, we define a filter omitting these trajectories (Fig.4, right / Fig.2).

 

 

 

4.png

Figure 2

 

 

Based on further information of the remaining object trajectories, which were obtained by graphical user interaction (Fig.3), the visualization tool allows us to remove also some false detections originating from highly variant video parts.

 

 

 

3.png

Figure 3

 

While exploring the video, it pointed out, that many trajectories are leading from people just crossing the scene or waiting at the pedestrian crossing (Fig. 4, left). We have also rejected these.

 

2.png

Figure 4

 

Finally we added our strongest hypotheses to the filter system, the interaction of multiple trajectories.

The trajectories of meeting people should begin with a merging stage then bend over to a stage of steady movement, before diverging in a trajectory split.

Filtering for one of these stages heavily decreases the amount of remaining trajectories. Fig. 5 depicts a splitting scenario. In this case a woman and man,

as described earlier in this section, meet each other, and leave in different directions after exchanging briefcases. This classification of the meaning of the scene

could only be done by human.

Since this is a very suspicious and Hollywood-like scene we rate it as the AAA-espionage event contained in this sequence. Further meeting of the woman involved in this encounter are also detected with this system.

 

5.png

Figure 5